Skip to content

fix: implement multi-replica distributed LockStore with Redis Lua scripts and monotonic fencing tokens (#378) - #386

Draft
s6pa1rta3n-lab wants to merge 1 commit into
Stellar-Analysis:mainfrom
s6pa1rta3n-lab:fix-issue-378
Draft

fix: implement multi-replica distributed LockStore with Redis Lua scripts and monotonic fencing tokens (#378)#386
s6pa1rta3n-lab wants to merge 1 commit into
Stellar-Analysis:mainfrom
s6pa1rta3n-lab:fix-issue-378

Conversation

@s6pa1rta3n-lab

Copy link
Copy Markdown

Summary

Resolves #378 by implementing a production-grade multi-replica LockStore backed by Redis (RedisLockStore) with atomic Lua scripts for acquisition, renewal, release, and fencing token validation.

Key Changes

  • Production RedisLockStore:
    • Implements the full LockStore trait backed by atomic Lua scripts executing single round-trip acquisitions, monotonic sequence increments, server-side TTL registrations, and stale write fencing.
    • Layered failover epoch protection (epoch) and optional WAIT replication acknowledgment guarantees cross-failover monotonicity without token reuse.
  • Fencing Token Generator:
    • Enhanced FencingTokenGenerator with epoch multipliers and update_if_greater store synchronization methods.
  • Chaos & Multi-Process Test Suite:
    • Extended lock_sigstop_test.rs to include a true multi-process test where a child process is paused with OS SIGSTOP, lock expires on Redis, secondary replica acquires with a higher token, and the resumed child's write is rejected with FencingTokenRejected.
    • Added test suites for simulated store failover monotonicity, concurrent acquisition storms, renewal-vs-expiry races, and store unavailability.
  • Failure Modes & Architecture Documentation:
    • Added comprehensive failure mode table and recovery matrix matching ttl_manager.rs style.

Acceptance Criteria Checklist

  • A LockStore implementation exists that is safe across ≥2 real, independent OS processes coordinating through a real backing store (RedisLockStore).
  • A test demonstrates that a paused-then-resumed holder's write, using its original fencing token, is rejected after a second replica has acquired the lock and issued a new token (test_multi_process_sigstop_fencing_rejection).
  • A test demonstrates monotonicity survives a simulated backing-store failover with no token reuse (test_failover_monotonicity_no_token_reuse).
  • The failure-mode table in ttl_manager.rs-style documentation is extended to cover this store, including what happens on backing-store unavailability mid-cycle.
  • Existing cargo test suite stays green (all 37 unit/lib tests and integration tests pass).

Payout Routing

  • EVM (Base/Arbitrum/Polygon/ETH): 0xF46C9F6d70C50BF81ef3588AB523a90a594a2F89
  • Stellar: GCL6OXAMLD75BMTINA6EMRUDWK5THQUSHMYNLSNBCJAPZJHNYJTUNIBC

…ripts and monotonic fencing tokens

- Add production RedisLockStore backed by atomic Lua scripts for acquisition, renewal, release, and fencing token validation
- Support failover epoch layering and WAIT replication acknowledgment to guarantee monotonicity across restarts and failovers
- Enhance FencingTokenGenerator with epoch multipliers and store synchronization
- Add comprehensive multi-process SIGSTOP chaos tests verifying stale writer rejection across independent OS processes
- Add test suites for simulated store failover monotonicity, concurrent acquisition storms, renewal-vs-expiry races, and store unavailability
- Document full failure-mode and recovery matrix matching ttl_manager.rs architecture
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Distributed lock fencing tokens are not actually distributed — no real multi-replica-safe LockStore exists

1 participant